Skip to content

fix(agentex-ui): abort the upstream request when the BFF client disconnects - #383

Merged
deepthi-rao-scale merged 3 commits into
mainfrom
erichwoo/bff-abort-upstream-on-disconnect
Jul 31, 2026
Merged

fix(agentex-ui): abort the upstream request when the BFF client disconnects#383
deepthi-rao-scale merged 3 commits into
mainfrom
erichwoo/bff-abort-upstream-on-disconnect

Conversation

@erichwoo-scale

@erichwoo-scale erichwoo-scale commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Problem

The /api/agentex catch-all BFF proxy streams SSE through unbuffered but passes no signal to fetch. When the browser goes away — closed tab, navigation, or the 300s Istio route timeout on the UI's virtualService — undici keeps the upstream request open. There is no timeout on the call either, so an orphan never expires, and the browser's EventSource then reconnects and opens a fresh one.

Each orphan leaves an agentex handler blocked in a Redis XREAD, pinning one connection from that pod's redis-py pool (REDIS_MAX_CONNECTIONS, default 200). They accumulate until the pools are exhausted, at which point agentex fails to serve new requests and only a restart recovers it.

Reported on Mayo Nonprod (Slack thread). At saturation Redis showed 1,094 connected_clients with 943 blocked_clients, while each of the two agentex-ui pods held ~570 established sockets to the agentex ClusterIP and served one inbound client apiece. Restarting only agentex-ui dropped backend Redis connections from ~1,094 to 279 without touching agentex, confirming the UI as the holder. Introduced in 0f07dc7 (#350).

Fix

Pass req.signal. App Router aborts it on client disconnect, which propagates through undici and destroys the upstream request — and makes the existing Istio route timeout an effective ceiling on upstream lifetime. A fixed timeout would be wrong here, since it would cut legitimate long-lived SSE streams.

The abort also rejects the in-flight fetch, so that case is answered with a 499 instead of propagating as a 500 — a departed client isn't an upstream failure. A genuine transport error still propagates exactly as before.

Verification

Built this branch, pointed AGENTEX_API_URL at a connection-counting SSE upstream, opened 5 streaming clients and killed them:

upstream live while connected 6s after clients killed
current main shape 5 5 ← leak
this branch 5 0

No ResponseAborted stack traces in the server log, server healthy afterward.

The main risk of adding a signal is truncating a response that is still streaming after the handler returns, so that was checked explicitly on a production build:

  • a 2 MB body streamed from upstream over ~2s arrived at exactly 2,048,011 bytes, 3/3 runs
  • a 666,669-byte streamed POST with duplex: 'half' echoed back intact

tsc --noEmit, eslint --max-warnings=0, and prettier --check are clean.

Follow-ups for the Agents team (not this PR)

Two agentex backend issues from the same thread remain open:

  1. cleanup_stream in the finally deletes a task-wide stream (streams_use_case.py:194). The topic is keyed by task id alone, so it is shared by every viewer. Worth landing with or right after this PR: today that finally almost never runs because disconnects don't propagate, and this change makes it run on every tab close. If something looks like a regression from this PR, it is that.
  2. SSE subscriptions have no terminal condition. A tab left open on a finished task still pins a pool connection indefinitely, polling XREAD every ~2.1s. The naive fix is unsafe: the client's retry loop (custom-subscribe-task-state.tsx, while (!signal?.aborted)) treats a clean server-side stream end as "reconnect immediately" — no backoff, no error-counter increment — so ending the stream server-side alone would produce a reconnect storm. Needs a coordinated client change.

Separately and unrelated to the leak: AGENTEX_API_URL falls back silently to http://localhost:5003, so a rename on either side yields a UI that can't reach the backend with no configuration error.

🤖 Generated with Claude Code

Greptile Summary

This PR fixes a connection-pool exhaustion bug in the Next.js App Router BFF proxy: upstream fetch calls were made without a cancellation signal, so client disconnects (tab close, navigation, Istio route timeout) left orphaned requests alive, each pinning a Redis connection in the backend's pool until restart.

  • Adds signal: req.signal to the upstream fetch so undici tears down the backend connection whenever the browser client disconnects or the Istio timeout fires.
  • Wraps the fetch in a try/catch and returns HTTP 499 for abort-triggered errors, using identity comparison against req.signal.reason (rather than error.name) to correctly distinguish a client-disconnect abort from a genuine transport failure that coincides with a disconnect — validated against Next.js's ResponseAborted internal abort path.

Confidence Score: 5/5

Safe to merge — a one-file, focused fix that adds a cancellation signal and handles the resulting abort error correctly.

The change is minimal and well-reasoned: it adds signal: req.signal to the upstream fetch and uses identity comparison against req.signal.reason to discriminate a client-abort from a coincident transport failure. The PR author verified the behavior against a production Next.js 15 build, confirmed no stream truncation on 2 MB bodies, and the comment in the code accurately documents why error.name would be wrong here. No other code paths are affected.

Files Needing Attention: No files require special attention.

Important Files Changed

Filename Overview
agentex-ui/app/api/agentex/[...path]/route.ts Adds signal: req.signal to upstream fetch and catches abort errors with identity comparison against req.signal.reason; logic is correct and well-commented.

Sequence Diagram

sequenceDiagram
    participant Browser
    participant NextBFF as Next.js BFF (route.ts)
    participant Upstream as Agentex Backend
    participant Redis

    Browser->>NextBFF: SSE request
    NextBFF->>Upstream: "fetch(target, { signal: req.signal, ... })"
    Upstream->>Redis: XREAD (blocking)
    
    alt Client disconnects (tab close / Istio timeout)
        Browser--xNextBFF: disconnect
        Note over NextBFF: req.signal aborted (ResponseAborted reason)
        NextBFF->>Upstream: AbortSignal propagated → undici cancels request
        Upstream--xRedis: connection released
        NextBFF-->>Browser: 499 (client already gone)
    else Genuine upstream transport error
        Upstream--xNextBFF: TypeError (ECONNREFUSED etc.)
        Note over NextBFF: error !== req.signal.reason → rethrow
        NextBFF-->>Browser: 500
    else Normal SSE stream
        Redis-->>Upstream: stream events
        Upstream-->>NextBFF: streamed response body
        NextBFF-->>Browser: unbuffered SSE
    end
Loading

Reviews (3): Last reviewed commit: "Merge branch 'main' into erichwoo/bff-ab..." | Re-trigger Greptile

…nnects

The `/api/agentex` catch-all proxy streams SSE through unbuffered but passes
no signal to `fetch`, so when the browser goes away — closed tab, navigation,
or the 300s Istio route timeout — undici keeps the upstream request open. Each
orphan leaves an agentex handler blocked in a Redis XREAD, pinning one
connection from that pod's pool until it is exhausted and agentex can no
longer serve requests. There is no timeout on the call either, so an orphan
never expires.

Pass `req.signal`, which App Router aborts on client disconnect, and answer
the resulting fetch rejection with a 499 so a departed client is not reported
as an upstream failure. A fixed timeout would be wrong here — it would cut
legitimate long-lived SSE streams.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@erichwoo-scale
erichwoo-scale requested a review from a team as a code owner July 29, 2026 00:35
Comment thread agentex-ui/app/api/agentex/[...path]/route.ts
Addresses review feedback: `req.signal.aborted` alone can mask a genuine
transport failure that happens to coincide with a client disconnect, since the
flag is already set by the time the catch runs.

Match on `error === req.signal.reason` instead. `fetch` rejects with the
signal's abort reason itself, so identity separates the two cases without
consulting the flag at all. Matching on `error.name === 'AbortError'` would not
work here: Next's abort reason is a `ResponseAborted`, and the class name is
minified in a production build, so both name checks fail and every disconnect
would rethrow as a 500.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
deepthi-rao-scale added a commit that referenced this pull request Jul 31, 2026
…; stop deleting shared stream (#385)

## What was broken

When you open a task in the UI, the backend keeps a live stream open to
push updates. That stream had **no way to end itself** — it only stopped
if the browser disconnected.

So when a task finished, the backend just kept listening forever,
holding one Redis connection each time. These pile up until the pool is
empty, at which point the pod can't serve requests (and its health check
fails, so it sits stuck until a manual restart).

A second bug: when one viewer left, the code deleted the task's
**shared** event stream — yanking it out from under everyone else
watching the same task.

## The fix

- **End the stream when the task is done.** The backend already gets a
`task_updated` event when a task finishes, so the stream now closes as
soon as it sees a terminal status.
- **Fallback for the rare cases** the event can't cover (viewer connects
after the task already finished, or the producer dies silently): a
lightweight check on the existing keepalive interval closes the stream
then too.
- **Stop deleting the shared stream** on one viewer's exit — the Redis
TTL already cleans it up on its own.

## Testing

Two integration regression tests (stream self-ends on terminal task; one
viewer disconnecting doesn't delete the shared stream).

**Verified live (A/B against `main`)** — same steps both times: create a
RUNNING task → open `GET /tasks/{id}/stream` → complete the task, while
watching Redis `blocked_clients`.

| | on task completion | Redis connection |
|---|---|---|
| **`main`** | stream keeps `XREAD`-looping; kept sending `:ping` 15s+
after completion and only ended when the client was force-disconnected |
`blocked_clients` stayed `1` — released only on client disconnect
(**zombie**) |
| **this branch** | stream delivers the terminal `task_updated` and
returns immediately (`Ending SSE stream …: received a terminal
task_updated event`) | `blocked_clients` `1 → 0` instantly, no client
disconnect needed |

On `main` the reader logged `Reading messages from Redis stream …` every
~2s indefinitely after the task was done; on this branch it stops the
moment the terminal event arrives. Same reproduction the incident
described, and the connection is released instead of pinned.

> Note: the unit/integration suite runs in CI; the tutorial-agent
integration jobs were red due to a pre-existing missing-`pytest-asyncio`
issue unrelated to this change (fixed separately in
scale-agentex-python).

Related: the UI-side half of this leak is #383.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- greptile_comment -->

<h3>Greptile Summary</h3>

This PR updates task SSE subscription lifecycle handling.
- Ends subscriptions when a terminal update is received, authoritative
task state becomes terminal, or the task disappears.
- Replays buffered events and closes when connecting to an
already-terminal task.
- Publishes failure updates and centralizes terminal/non-terminal status
classification.
- Preserves shared Redis streams when individual subscribers disconnect.
- Adds integration coverage for terminal shutdown, late connections,
reclaimed keys, transient lookups, and shared-stream retention.

<details><summary><h3>Confidence Score: 4/5</h3></summary>

The PR should not merge until recoverable FAILED tasks no longer
terminate subscriptions that must observe their subsequent RUNNING
updates.

The stream returns upon receiving any FAILED task update, while the task
service intentionally supports forwarding that same FAILED task again
and publishing a RUNNING recovery update; viewers whose subscriptions
closed on FAILED cannot receive that recovery or subsequent events.

**Files Needing Attention:** agentex/src/domain/entities/tasks.py,
agentex/src/domain/use_cases/streams_use_case.py,
agentex/src/domain/services/task_service.py
</details>

<details><summary><h3>Important Files Changed</h3></summary>

| Filename | Overview |
|----------|----------|
| agentex/src/domain/entities/tasks.py | Introduces canonical terminal
and non-terminal status sets shared by state transitions and SSE
termination. |
| agentex/src/domain/services/task_service.py | Publishes FAILED updates
through the normal update path, while existing retry behavior can still
restore FAILED tasks to RUNNING. |
| agentex/src/domain/use_cases/streams_use_case.py | Adds event-driven
and authoritative fallback termination and stops deleting the shared
Redis topic, but the previously reported recoverable-FAILED lifecycle
conflict remains. |
| agentex/src/domain/use_cases/tasks_use_case.py | Reuses the canonical
non-terminal status set for transitions to terminal states. |
| agentex/tests/integration/test_task_stream.py | Adds regression
coverage for terminal shutdown, late subscribers, vanished tasks,
transient lookups, reclaimed keys, and shared-stream preservation. |

</details>

<details><summary><h3>Sequence Diagram</h3></summary>

```mermaid
sequenceDiagram
    participant Client
    participant SSE as StreamsUseCase
    participant DB as Task Store
    participant Redis
    Client->>SSE: Subscribe to task stream
    SSE->>Redis: Snapshot stream cursor
    SSE->>DB: Read authoritative task state
    alt Task already terminal
        SSE->>Redis: Replay buffered events
        SSE-->>Client: Events, then close
    else Task is non-terminal
        loop Until terminal, missing, or disconnect
            SSE->>Redis: Read new events
            alt Terminal task_updated received
                SSE-->>Client: Terminal event
                SSE-->>Client: Close stream
            else Status-check interval elapsed
                SSE->>DB: Recheck task
                alt Terminal or missing
                    SSE-->>Client: Close stream
                end
            end
        end
    end
    Note over SSE,Redis: Subscriber exit does not delete the shared stream
```
</details>

<sub>Reviews (14): Last reviewed commit: ["fix(agentex): terminate SSE
task
streams..."](85dfe6b)
| [Re-trigger
Greptile](https://app.greptile.com/api/retrigger?id=48824816)</sub>

<!-- /greptile_comment -->

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@deepthi-rao-scale
deepthi-rao-scale enabled auto-merge (squash) July 31, 2026 17:24
@deepthi-rao-scale
deepthi-rao-scale merged commit c729b12 into main Jul 31, 2026
14 checks passed
@deepthi-rao-scale
deepthi-rao-scale deleted the erichwoo/bff-abort-upstream-on-disconnect branch July 31, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants